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DETAILED ACTION 
Response to Amendment 

1 . In response to the office action from 3/30/2007, the applicant has submitted an 
amendment, filed 7/2/2007, amending independent claims 1, 9, and 22, while arguing to traverse . 
the art rejection based on the added limitation regarding selectively enabling an input modality 
based on the state of a dialog workflow description and then controlling an output prompt based 
on the enabled input modality (Amendment, Pages 17-20), The applicant's arguments have been 
fully considered, but are moot with respect to the new grounds of rejection necessitated by the 
amended claims and in view of Bourguet et al (U.S. Patent Application Publication: 
2002/0178344). 

2. Due to amended claims 7 and 1 0, the examiner has withdrawn the previous claim 
objection directed to minor informalities. 

3. . Due to the amendment of claims 9 and 22, the examiner has withdrawn the previous 35 
U.S.C. 112, first paragraph rejections. 

4. In light of the applicant's arguments regarding amended claims 1 and 9 (Amendment, 
Page 14), the examiner has withdrawn the previous 35 U.S.C. 101 rejection directed to these 
claims and their associated dependent claims. The examiner notes that although amending claim 
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22 to include a "tangible" computer readable medium overcomes an aspect of the previous 35 
U.S.C. 101 rejection directed to this claim and its associated dependent claims, the term 
"tangible" is not defined in the specification, thus raising an issue of new matter (see the below 
35 U.S.C. 1 1 2, first paragraph rejection). Also, the steps claimed are still do not include the 
program execution by a computer, thus, the previous 35 U.S.C. 101 rejection is maintained for 
this reason. To overcome this rejection the examiner recommends changing "one or more 
processors" to -one or more computer processors- (supported in the specification at Page 70, 
Lines 13-14). 

Double Patenting 

5. The nonstatutory double patenting rejection is based on a judicially created doctrine 
grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or 
improper timewise extension of the "right to exclude" granted by a patent and to prevent possible 
harassment by multiple assignees. A nonstatutory obviousness-type double patenting rejection 
is appropriate where the conflicting claims are not identical, but at least one examined 
application claim is not patentably distinct from the reference claim(s) because the examined 
application claim is either anticipated by, or would have been obvious over, the reference 
claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re 
Goodman, 1 1 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 
USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re 
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Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); and In re Thorington, 418 F.2d 528, 163 
USPQ644(CCPA 1969). 

A timely filed terminal disclaimer in compliance with 37 CFR 1 .321(c) or 1 .321(d) may 
be used to overcome an actual or provisional rejection based on a nonstatutory double patenting 
ground provided the conflicting application or patent either is shown to be commonly owned 
with this application, or claims an invention made as a result of activities undertaken within the 
scope of a joint research agreement. 

Effective January 1, 1994, a registered attorney or agent of record may sign a terminal 
disclaimer. A terminal disclaimer signed by the assignee must fully comply with 37 CFR 
3.73(b). 



6. Claims 1-25 are provisionally rejected on the ground of nonstatutory obviousness-type 
double patenting as being unpatentable over claims 1-16 and 48-49 of copending Application 
No. 10/61 7,422. Although the conflicting claims are not identical, they are not patentably 
distinct from each other because both claimed inventions are directed to the same system/method 
for executing a multimodal software application utilizing input and output peripheral devices, 
workflow descriptions based on the software application, and interface components. Although 
App. No. 10/617,422 omits the prompt control of the presently claimed invention, the omission 
of such an element would be obvious to one of ordinary skill in the art because both systems 
essentially perform the same steps of providing input data, processing that data according to a 
workflow, and providing output data (i.e., a prompt) in response. 
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This is a provisional obviousness-type double patenting rejection because the conflicting 
claims have not in fact been patented. 

Claim Rejections - 35 USC § 101 

7. 35 U.S.C. 101 reads as follows: 

Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or 
any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and 
requirements of this title. 

8. Claims 22-25 are rejected under 35 U.S.C. 101 because the claimed invention is directed 
to non-statutory subject matter. 

Claim 22 is drawn to program "instructions" as recited in the preamble and as such is 
non-statutory subject matter. Data structures not claimed as encoded in computer readable media 
are descriptive material per se and are not statutory because they are not capable of causing 
functional change in the computer. See, e.g., Warmerdam, 33 F.3d at 1361, 31 USPQ2d at 1760 
(claim to a data structure per se held nonstatutory). Such claimed data structures do not define 
any structural and functional interrelationships between the data structure and other claimed 
aspects of the invention, which permit the data structure's functionality to be realized. In 
contrast, a claimed tangible computer readable medium encoded with a data structure defines 
structural and functional interrelationships between the data structure and the computer software 
and hardware components which permit the data structure's functionality to be realized, and is 
thus statutory. Similarly, computer programs claimed as computer listings per se, i.e., the 
descriptions or expressions of the programs are not physical "things." They are neither computer 
components nor statutory processes, as they are not "acts" being performed. Such claimed 
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computer programs do not define any structural and functional interrelationships between the 
computer program and other claimed elements of a computer, which permit the computer 
program's functionality to be realized. To overcome this rejection the examiner recommends 
changing "one or more processors" to -one or more computer processors- (supported in the 
specification at Page 10, Lines 13-14). 

Dependent claims 23-25 fail to overcome the 35 U.S.C. 101 rejection directed towards 
claim 22, and thus, are also directed to non-statutory subject matter by virtue of their 
dependency. 



Claim Rejections - 35 USC § 112 

9. The following is a quotation of the first paragraph of 35 U.S.C. 1 12: 

The specification shall contain a written description of the invention, and of the manner and process of making 
and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it 
pertains, or with which it is most nearly connected, to make and use the same and shall set forth the best mode 
contemplated by the inventor of carrying out his invention. 

10. Claims 22-25 are rejected under 35 U.S.C. 1 12, first paragraph, as failing to comply with 
the written description requirement. The claim(s) contains subject matter that was not described 
in the specification in such a way as to reasonably convey to one skilled in the relevant art that 
the inventor(s), at the time the application was filed, had possession of the claimed invention. 

Specifically, claims 22-25 recite a "tangible computer-readable storage medium, " which 
is not disclosed in the specification. Although the specification discloses a number of computer 
readable mediums, which can be considered as being directed to a tangible medium (see Page 
10), the term "tangible" is not used in the specification to define these mediums. Thus, claims 
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22-25 fail to comply with the written description requirement. It is additionally pointed out, 
however, that the tangible mediums disclosed in the specification are defined by the term 
"recordable" media (Page 10, Line 22), thus, it is recommended that the claims be amended to 
recite -computer-readable recordable medium- in order to overcome this 35 U.S.C. 112, first 
paragraph rejection. 



Claim Rejections - 35 USC § 103 

1 1 . The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

12. Claims 1-3, 7-9, 11-12, 15, and 19-20 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Morin et al (U.S. Patent: 5,748,841) in view of Bourguet et al (U.S. Patent 
Application Publication: 2002/0178344). 

With respect to Claim 1, Morin discloses: 

A first set of peripheral devices for receiving inputs from a user in a plurality of 
modalities and a second set of peripheral devices for generating outputs to a user in a plurality of 
modalities (various input and output devices, Col. 11, Line 40- Col. 12, Line 15; and Fig. 1)\ 

The multimodal software application, wherein said multimodal software application 
receives first data, that is reflective of an input, from a first set of peripheral devices and output 
second data, to form an output, to a second set of peripheral devices (multimodal target software 
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application capable of receiving data from a set of input devices and outputting data from a set 
of output devices, Col 13, Lines 30-67; and various input and output devices, Col 11, Line 40- 
Col 12, Line 15); 

A dialog engine in communication with the multimodal software application, wherein 
said dialog engine executes a workflow description, having a plurality of dialog units that form a 
dialog, received from the multimodal software application and provide the first data to the 
multimodal software application (dialog manager, having a plurality of dialog instruction steps, 
that executes dialog tasks in accordance with an application script and provides input data to an 
application, Col 9, Line 1- Col 10, Line 25); 

Said dialog engine further controlling outputting of a prompt from the workflow 
description based on an input state of the first set of peripheral devices (dialog manager utilizing 
context and history to monitor an input state, Col 10, Lines 1-25 and generating request and 
other dialog prompts, Col 9, Lines 1-57; and output devices, Col 8, Lines 17-37); and 

A respective interface component associated with each peripheral device within said first 
and second sets; wherein each interface component is configured to provide the second data, if 
any, to the associated peripheral device and receive the first data, if any, from the associated 
peripheral device (I/O device drivers that provide input output data to their associated devices, 
Col 6, Lines 5-48; Col 7, Lines 17-27; and Col 8, Lines 17-37). 

Although Morin recites the ability to generate an output prompt based on the state of 
various multimodal inputs (Col 9, Line /- Col 10, Line 25) and further notes enabling different 
multimodal I/O devices for different dialog tasks (dialog dependent devices, Col 13, Lines 55- 
61), Morin does not specifically suggest selectively enabling an input modality from a set of 
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peripheral devices based on a state within the dialog and then outputting a prompt based on an 
input state of the selectively enabled multimodal input devices. Bourguet, however, recites a 
dialog manager in a multimodal interactive system (Figs. 1-2) that dynamically chooses input 
modalities available to a user based on a current dialog state (Paragraphs 0083 and 0089). 
Bourguet further notes that the states of the available input modalities at a particular dialog state 
are utilized to generate an output prompt (Paragraph 0088). 

Morin and Bourguet are analogous art because they are from a similar field of endeavor 
in multimodal user interface systems. Thus, it would have been obvious to a person of ordinary 
skill in the art, at the time of invention, to modify the teachings of Morin with the means for 
selectively choosing available input modalities based on a dialog state as taught by Bourguet in 
order to provide a multimodal user interface that is capable of dynamically selecting modalities 
that are expected by a particular dialog (Paragraph 0083). 

With respect to Claim 2, Morin further discloses: 

The control includes interrupting the prompt if the first data is received while the prompt 
is being output ("talk over" feature, Col. 12, Lines 1-15). 
With respect to Claim 3, Morin further discloses: 

Delaying outputting of the prompt if one of the first set of peripheral devices is receiving 
the first data (dialog turns that require the dialog manager to wait for a complete user response 
before taking its turn, Col. 9, Lines 35-56). 

With respect to Claim 7, Morin further discloses: 
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The first set of peripheral devices includes one or more of a speech recognition system, a 
radio-frequency identifier scanner, a bar code scanner, a touch screen, a keypad, and a computer 
(input devices, Col. 7, Lines 17-27). 

With respect to Claim 8, Morin further discloses: 

The second set of peripheral devices includes one or more of a voice synthesis system, a 
display screen and a computer (output devices, Col. 8, Lines 17-37). 
With respect to Claim 9, Morin discloses: 

Executing a workflow description received from the multimodal application, said 
workflow description including a plurality of dialog units that form a dialog (application dialog 
script, which defines task dialog steps, executed by a dialog manager, Col. 9, Line 1- Col. 10, 
Line 25; and Col 14, Lines 1-23); 

Receiving inputs from a user in at least one modality from at least one peripheral device 
of a plurality of peripheral devices and generating outputs to a user in at least one modality with 
at least one peripheral device from another plurality of peripheral devices (various input and 
output devices, Col. 11, Line 40- Col. 12, Line 15; and Fig. 1); 

Outputting a prompt of a first workflow object via a peripheral device of the another 
plurality of peripheral devices, said prompt related to the multimodal application (outputting task 
requests or messages to a user, Col. 9, Lines 35-56; Col. 12, Lines 1-15); and 

Controlling the outputting of the prompt according to an input state of the plurality of 
peripheral devices (dialog manager utilizing context and history to monitor an input state, Col. 
10, Lines 1-25 and generating request and other dialog prompts, Col. 9, Lines 1-57; and output 
devices, Col. 8, Lines 17-37). 
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Although Morin recites the ability to generate an output prompt based on the state of 
various multimodal inputs (Col 9, Line 7- Col 10, Line 25) and further notes enabling different 
multimodal I/O devices for different dialog tasks (dialog dependent devices, Col 13, Lines 55- 
61), Morin does not specifically suggest selectively enabling an input modality from a set of 
peripheral devices based on a state within the dialog and then outputting a prompt based on an 
input state of the selectively enabled multimodal input devices. Bourguet, however, recites a 
dialog manager in a multimodal interactive system (Figs. 1-2) that dynamically chooses input 
modalities available to a user based on a current dialog state (Paragraphs 0083 and 0089). 
Bourguet further notes that the states of the available input modalities at a particular dialog state 
are utilized to generate an output prompt (Paragraph 0088). 

Morin and Bourguet are analogous art because they are from a similar field of endeavor 
in multimodal user interface systems. Thus, it would have been obvious to a person of ordinary 
skill in the art, at the time of invention, to modify the teachings of Morin with the means for 
selectively choosing available input modalities based on a dialog state as taught by Bourguet in 
order to provide a multimodal user interface that is capable of dynamically selecting modalities 
that are expected by a particular dialog (Paragraph 0083). 

With respect to Claim 11, Morin discloses the "talk over" feature, as applied to Claim 2. 

With respect to Claim 12, Morin further discloses: 

The step of outputting includes outputting an audio prompt (spoken messages, Col 12, 
Lines 7-75); and 

The step of receiving includes receiving voice data from a speech recognition system 
(speech recognizer, Col 7, Lines 28-48). 
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With respect to Claim 15, Morin further discloses: 

Performing the step of terminating if the data is received from a predetermined peripheral 
device; and omitting the step of terminating if the input is received from other than the 
predetermined device ("talk over" feature that interrupts a spoken prompt in response to a 
speech input and would inherently omit prompt cancellation if any other type of input was 
received, Col 12, Lines 1-15; and Col 7, Lines 28-32). 

Claim 19 contains subject matter similar to Claim 3, and thus, is rejected for the same 
reasons. 

With respect to Claim 20, Morin further discloses: 

The step of delaying includes the steps of: delaying outputting the prompt to one of the 
another plurality of peripheral devices; and permitting outputting the prompt without delay to 
another of the plurality of peripheral devices (simultaneous output of text and speech and 
canceling only speech messages using a "talk-over" feature, Col 12, Lines 1-15). 

13. Claims 4-5 and 16-18 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Morin et al in view of Bourguet et al and further in view of Gergic et al (U.S. Patent Application 
Publication: 2002/0198719). 

With respect to Claims 4-5 and 16-17, Morin in view of Bourguet discloses the 
multimodal system and method featuring prompt control based on an enabled input modality for 
processing input/output data, as applied to Claims 1 and 9. Although Morin discloses the ability 
to handle successive user responses (Col 12, Lines 41-54) and request a user response to fill-in 
missing information (Col 9, Lines 35-56), Morin in view of Bourguet does not specifically 
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suggest processing a user input data directed to two prompts, which would result in an avoided 
subsequent prompt. Gergic, however, discloses the ability of a user to fill two fields of missing 
information with a single utterance in response to an initial dialog, which does not require the 
output of a subsequent dialog prompt (Paragraph 0121). 

Morin, Bourguet, and Gergic are analogous art because they are from a similar field of 
endeavor in interactive multimodal dialog systems. Thus, it would have been obvious to a 
person of ordinary skill in the art, at the time of invention, to modify the teachings of Morin in 
view of Bourguet with the ability of a user to fill two fields of missing information taught by 
Gergic in order to provide a means for supporting parallel activation of dialog objects (Gergic, 
Paragraph 0104) and allow for more efficient user/machine interaction. 

With respect to Claim 18, Gergic further recites: 

The data relates to the first workflow object and a plurality of other workflow objects 
(filling more than one field, Paragraph 0121). 

14. Claims 6 and 14 are rejected under 35 U.S.C. 103(a) as being unpatentable over Morin et 
al in view of Bourguet et al and further in view of Hanle et al (U.S. Patent: 5,012,511). 

With respect to Claims 6 and 14, Morin in view of Bourguet discloses the multimodal 
system and method featuring prompt control based on an enabled input modality and device 
drivers for processing input/output data, as applied to Claims 2 and 1 1 . Morin in view of 
Bourguet does not teach the use of non-interruptible prompts, however Hanle discloses the use of 
such prompts in an interactive dialog system (Col. 9, Lines 33-51). 
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Morin, Bourguet, and Hanle are analogous art because they are from a similar field of 
endeavor in interactive voice response systems. Thus, it would have been obvious to a person of 
ordinary skill in the art, at the time of invention, to modify the teachings of Morin in view of 
Bourguet with the non-interruptible prompts taught by Hanle in order to force the user to listen to 
important prompts, such as error prompts (Hanle, Col. 9, Lines 50-51). 

1 5. Claims 10, 22-23, and 25 is rejected under 35 U.S.C. 103(a) as being unpatentable over 
Morin et al in view of French-St. George et al (U.S. Patent: 6,012,030). 

With respect to Claims 10, Morin in view of Bourguet discloses the multimodal system 
and method featuring prompt control based on an enabled input modality and device drivers for 
processing input/output data, as applied to Claim 9. Although Morin teaches the concept of 
speech control of computer applications (Col. 13, Lines 61-67), Morin in view of Bourguet does 
not explicitly disclose prompts relating to visual control of a GUI screen, however French-St. 
George discloses such prompts (Col. 7, Line 37- Col. 8, Line 18). 

Morin, Bourguet, and French-St. George are analogous art because they are from a 
similar field of endeavor in interactive multimodal dialog systems. Thus, it would have been 
obvious to a person of ordinary skill in the art, at the time of invention, to modify the teachings 
of Morin with the generation of prompts relating to visual control of a GUI screen as taught by 
French-St. George in order to provide a means for dynamically adjusting audio prompts based on 
a user's interaction (French-St. George, Col. 3, Line 64- Col. 4, Line 2). 

Claim 22 contains subject matter similar to Claims 9 and 10, and thus, is rejected for the 
same reasons. Morin further discloses method implementation at a server computer (Col. 5, 
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Lines 43-56), which would inherently require some type of stored instruction set for method 
execution. 

With respect to Claim 23, Morin discloses the "talk over" feature, as applied to Claim 2. 
Claim 25 contains subject matter similar to Claim 3, and thus, is rejected for the same 
reasons. 

1 6. Claim 21 is rejected under 35 U.S.C. 1 03(a) as being unpatentable over Morin et al in 
view of Bourguet et al and further in view of Setlur et al (U.S. Patent: 5,956,675). 

With respect to Claim 21, Morin in view of Bourguet discloses the multimodal system 
and method featuring prompt control based on an enabled input modality and device drivers for 
processing input/output data, as applied to Claim 19. Morin in view of Bourguet does not teach 
analyzing if a user's input relates to a prompt to identify a valid barge-in attempt, however, 
Setlur recites executing a barge-in attempt only when a speech input is related to an expected 
aural prompt response (Col 5, Lines 1-23). 

Morin, Bourguet, and Setlur are analogous art because they are from a similar field of 
endeavor in interactive voice response systems. Thus, it would have been obvious to a person of 
ordinary skill in the art, at the time of invention, to modify the teachings of Morin in view of 
Bourguet with the barge-in identification means taught by Setlur in order to provide a means for 
improved barge-in reliability (Setlur, Col. 5, Lines 18-20). 
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1 7. Claim 24 is rejected under 35 U.S.C. 1 03(a) as being unpatentable over Morin et al in 
view of Bourguet et al in view of French-St George et al and further in view of Gergic et al (U.S. 
Patent Application Publication: 2002/0198719). 

With respect to Claim 24, Morin in view of Bourguet and further in view of French-St. 
George discloses the multimodal system and method featuring prompt control based on an 
enabled input modality and device drivers for processing input/output data, as applied to Claim 
22. Although Morin discloses the ability to handle successive user responses (Col. 12, Lines 41- 
54) and request a user response to fill-in missing information (Col. 9, Lines 35-56), Morin does 
not specifically suggest processing a user input data directed to two prompts, which would result 
in an avoided subsequent prompt. Gergic, however, discloses the ability of a user to fill two 
fields of missing information with a single utterance in response to an initial dialog, which does 
not require the output of a subsequent dialog prompt (Paragraph 0121). 

Morin, Bourguet, French-St. George, and Gergic are analogous art because they are from 
a similar field of endeavor in interactive multimodal dialog systems. Thus, it would have been 
obvious to a person of ordinary skill in the art, at the time of invention, to modify the teachings 
of Morin in view of Bourguet and further in view of French-St. George with the ability of a user 
to fill two fields of missing information taught by Gergic in order to provide a means for 
supporting parallel activation of dialog objects (Gergic, Paragraph 0104) and allow for more 
efficient user/machine interaction. 
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Conclusion 

1 8. Applicant's amendment necessitated the new ground(s) of rejection presented in this 
Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). 
Applicant is reminded of the extension of time policy as set forth in 37 CFR 1 .136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1 . 1 36(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the date of this 
final action. 

19. The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure: 

Maes (U.S. Patent; 7,21 6, 35 J)- discloses a system capable of enabling different input 
modalities. 

20. Any inquiry concerning this. communication or earlier communications from the 
examiner should be directed to James S. Wozniak whose telephone number is (571) 272-7632. 
The examiner can normally be reached on M-Th, 7:30-5:00, F, 7:30-4, Off Alternate Fridays. 
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If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Patrick Edouard can be reached at (571) 272-7603. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 



James S. Wozniak 
9/10/2007 




